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ABSTRACT / ' 

Whether coaching for standardized admission tests 
such as the Scholastic aptitude Test fSATJ is effective was 
investigated- The Bureau of Consumer Protection of the Federal Trade 
Commission evaluated the effectiveness of two commercial coaching 
schools that offered preparation courses for the SAT* The researchers 
concluded that one of the two schools raised its students' scores an 
average o? approximately 25 points each on both verbal and 
mathematical sections of the test* <ataff found that a number of *other 
studies, some funded by the College Board fCB) which sponsors tfte 

also shown that students coYild make significant gains on the 
result of participating in a coaching. program- Staff also 
the descriptiv-e materials regarding the $AT which are 
provided to students and schools each year by Educational Testing 
Service and! CB« Staff concluded that although the harsh criticism of 
the tBf coaching that had been levied ip the late sixties had been 
modified in more recent materials, even the 1979*80 materials did not 
appear to recognize the possibility of meaningful score gains through 
coaching. .The CB has indicated fts intention to revise descriptive 
materials iir the future. fEL) 
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The views contained in this document are those 

of the .staff of the Federal Trade Commission, The ; 

views have not been , formally adopted by the Coromissio 



S tandar&dized admission test£ have Recently become^the sub 
ject of public debate. Critics "have questioned whether" the 
t tests can accurately predict future academic achievement and 
whe the r the - tests un fa i t ly disc r imina te* against the poor, 
minorities and culturally disadvantaged students. Others have 

4 

also questioned the use of multiple-choice tests in taking ad- 
missions decisions because such tests carfcnot reflect-e 
'student's creativity or motivatron. ( t I 

The Boston Regional Office of . the FedefaT Trade Commis- 
sion has been investigating one important issue regard ing 
standardized admission^ tests whether coaching for s^ch tests 
is effective. The investigation was spawned by a concern that 
commercial coachitig schools were making unfair and deceptive , 

claims regarding effectiveness. These schools/ claimed that 

/ 

their programs could help students improve, th^fir scores on a 
variety of standardized^ tests* by as much as ope hundred 
ppints , At the sam£ time t however , thre Educational Testing 

Service ("ETS"), the largest o^ the testmakeV?, maintained 

' * - " ' * / " . * 

that coacKing would do little.to help, / 

' - ■/''■' 

As' part of the investigation f the Bure/au of Consumer . Pro- 

\ - / 

tection of the Federal trade Commission evaluated the effec- 
tiveness -of two . commercial cdaching schools that, offered 
preparation' courses for the Schol?!stic Aptitude Testf '{ "SAT" )., 
The* researchers concluded that one of the two schools was ef- 



fecti'.ve in* raising its students 1 scores an\ average of approxi- 
mately twenty^five points each, on botti the verfcfal and mathemat- 
ical Sections of'the test- These findings led the staff of 
the Boston Regional Office to review other* research op 
coaching for the SAT** Staff found that a number of other 
studies -some funded by the sponsor of the'SAT itself, the Col- 
lege Board - had also shown' that students could ma(ce signifi- 
cant gains on the SAT as a result; of participating in a 
coaching program* t 

The question of whether coaching for the SAT is effective 
is an important one because of Ehe role that the SAT plays in 
college admissions decisions* In a recent survey, 42% of *the 
private four year colleges Jand 37*9% of the public four .year 
col£ege£ responding indicated that they had minimum SAT score 
requirements below which applicants generally are not 
considered eligible for admission* In light of the emphasis 

that students, high schools, colleges and even ^communities 

* * 

place on the results of the; SAT, Bostonr Regional Office staff 
reviewed the descriptive materials regarding the SAT which are 
presided to students and schools each year by ETS and College 
Board* Staff wanted to determine whether the materials fdlly 
and accurately described the possible benefits -of coaching* As* 
a result of that review and ^discussions with ETS and College 
Board, the Boston Regional Office concluded that there was 
cause for concern* Though the harsh Criticism 6f coaching*" 



thath§d been levied in the late sixties had been modified in 
more 'recent materials, even the 1979*80 materials did not ap- * 
pear to recognize , the possibility of meaningful score gains 
through coaching* V V * ' - , ■ *■ 

The Commission shared the Boston Regional Office's con- 
cefn? about the 1979*80, materials and offered both ,ETS and Col 
lege ■ poard an opportunity to respond to those concerns. ETS 
Submitted a lengthy report on coaching which included two sta- 
tistical ieanalys/s of; the Bureau of Consumer Protection's 
.study « The^* ETS r eanal yses largely confirmed the f indings* made 
by the Bureau* ' 

, ■ The College , Board submitted the descriptive materials on t 
the SAT prepared f6c the 1980-81 testing year, The^ mater ial s 

included'a one page message on coaching sent to .high schools 

* • 1 

which, <for t;he first time, prQvides a morp comprehensive expla 

nation, of the possible benefits of coaching* The message^ rec- 
ognizes that there are dif ferent^types oL coaching programs 
and acknowledges L that different results may occur* The mes- 

• / +s ; - - - * / * - - 

s^geyexplains that tome- programs had been found to produce * 
*g^iO$ of 'about ip points on the verbal section and 15 points 
/ on -the^m^hematical section, whiYe others had resulted 

ir\ increases as high as 25-30/points on a, section of 
'the S^T* While the message may not reflect the still greater 

incr^ea^es found in some^ studies of coachi'ng, it represents 

a if. important ru^Oe toward full disclosure* The other more ■ 



in 



traditional materials on the SAT, which include a^gtiide 

and bulletin f 0r students and a guide for* high school counselors 

and admissions officers, do not parallel the new message. 

These materials appear to recognize the findings made in 

a limited number of studies arid acknowledge orily" the 10 

and 15*point gains from coaching* The materials do not 

reflefct the fact that some types of" coaching c£n. result 

in greater -gains. 'By 1-etter to staf f of the Commission, 

College Board has indicated its intention to revise such 

descriptive materials in the fature.' 

A finding that coaching can be effective for £ 
standardized -admission fcest'such as the SAT has far-reaching 
educational implications^ Questions are raised about the mean- 
"ing.of an examination "designed to measure verbal and mathem^ti- 
cal abil i ties said to be developed over a li f etime of learning 
if scores can be meaningfully changed in" a few week's -or 

* * 

months . Oue st ions of fai rness are also raised if * go aching cai> 

i % * 

have a substantial impact on scores but suctr coaching' is 
not equally available to all- students', particularly the ■ 
less affluent. Much research remains'to b^'done before 

these questions" can begin to be addressed* More must' be *,« 

/ c ■ - , 

known about what precisely the SAT m^Sures anjd why coaching 

can help students achieve meaningful score gains'. '^otfe 

must be known about the availability o£ coaching;. Md-re 

must be learned about the kinds "of students who lifray^benef i E, 



the most ftom £ coaching program and what impact unequal 

access to coaching may have oil students 1 access to higher 

* * * * 
education , . 

* • ■+ 
As a law enforcement agency charged with protecting the 

public from unfair and deceptive practices, the Federal Trad.e 

* * * 

Commission has concentrated its efforts on assuring that orga 

^niaations involved in the ^standardized testing industry make 

full and * f^ir disclosure r egard ing the coaching issue . The * 

educational policy questions raised by the Federal Trade 

Commission' s investigation into the coaching issue are ones 

which should be addressed by those with the mandate and 

greater expertise ir; this important area , 



I, Introduction > 

+ Standardized admission tests haVe .recently become the 
focus of much attention* The validity and value of such tests 
has been t # he subject of public debate. Congress , as well as 
many state legislatures, has been considering disclosure leg- 
islation regarding these tests. The Federal Trade Commission 
has conducted- an investigation into one important is£ae 
regarding standardized admission .tests -the effectiveness 
of special ^preparation for such test's* Because of the rftfe 

, that standardized admission tests play in the educational 
lives of young people and in light of the ser ious "educational 
policy questions r^^ed by the coaching is^ue, the Boston 
Reg ional Off ice bel ieves that it , is important to fully 
report on the Federal Trade Commission's investigation, 

•The views contained ift this document are those of ,the staff 
of the Federal Trade Commission* The yiews have ftot been 
forraally adopted by the Commission* 

In late,1975 the Boston Reg ional h Office of the Federal 
Trade Commission was authorised to conduct ein industry- 
wide investigation into the validity of claims made regarding 
the effectiveness of coaching for standardized admission 
tests,. The investigation was spawned by concern that commercial 
coaching' schools were misleading prospective students by 
unfair and deceptive claims of effectiveness* The desire 
to assess those claims resulted in the undertaking of- a 
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detailed statistical analysis' of the effectiveness of commercial 

coaching school programs by the Federal Trade Commission's 

■ * J 

Bureau of Consumer Protection. That study concluded tftat 

fox at least some students coaching for the Scholastic 

Aptitude Test could be effective, A revie^w' of other research 

on the effectiveness of coaching for the Scholastic Ap,titude* 

Test made- clear that the Bureau * s >pos i t ive find ing s were ' 

hot an isolated occurrence . Other studies had reached,. 

similar cone 1 us ions about the effectiveness of coaching. 

The Bureau's positive findings raised concern as to 

Whether the possible effects of coaching for the Scholastic 

Aptitude Test were properly being* reflected in the materials 

wh ich the test maker , Educat ipnal Test ing Serv ice , and the 

test sponsor , College Board , prov ided to students 1 and educators , 

m 

The Bostorr Regional Office's review of those materials 

/ 

revealed that intense criticism pf coaching had been levied - 
in the materials distributed An the late sixties. While 
that' criticism was tempered over the ensuing years, the 
overall message remained that' coaching for the Scholastic* 
Aptftude Test was unlikely to be of any significant . benefit, 
A 1980 messaged coaching sent to secondary schools by 

the College Board provides, for the first time, a tnore 

* 9 . * 

comprehensive discussion of the possible benefits of 

various forms of coaching for the Sct\plastic Aptitude 

Test, Similar disclosures have not yet been made, in -other 



student and educator^ descriptive materials onf the Scholastic 
Aptitude Test, although the College Board has indicated 
'that 1981-82 materials will- contain revised , information " 
about coaching*- 

" A finding that coaching can he effective 'for a standardized 
admission test such. as the Scholastic Aptitude Test raises 
a number of very important educational pol ifcy questions . <^ 
Though the Federal Trade ..Commi ssion has committed its resources 
ta assuring that organizations involved in the standardized 
testing industry make full and fair disclosure regarding 
the" coaching issue , it has neither th^ mandate nor the 
expertise to deal with all of the important educational * 
questions* Staff has includedra discussion of some of those 
^questions in this report so that: the expertise^ and resources 
of the appropriate government agencies, as welVas private 
researchers, may be directed to finding the answers^* 
II * Background on the Standardized Admission frest Industry 
A* Standardized Admission Tests . * . . 

Standardized admission/ tests are professionally developed 
examinations, often administered nationw^e and most often ijj 
multiple-choice format, which are used in the evaluation of ap- * 
plicants for admission to a wi*de variety of. educational 
insti tut ions, including private' secondary schools, coll eg es 
and graduate schools* Each year, over a million students take 
some form of standardized admission test* 



^ ' There are a number of both prof it. and nonprofit corpora- * 

tions engaged in the development of standardized admission" \ 

tests. The tests "involved dn our investigation were developed 

by the Educational Testing Service ("ETS"),. ETS , a nonprofit 

corporation located in Princeton, New Jersey, with reported 

Revenues of ov^r ninety w ill ion dollars for its 1979 ^fiscal 

year , 1 is^the largest te£tmaker and administrator 6f 

standardised admission tests in the nation. Its tests 

include. the Secondary School Admission Test, use>§ for grades 

* * 

six through eleven, the 4 Scholastic Apt itude Test ("SAT"), < • 
used by colleges' and universities throughout^ the country 
th£ Law School Admisjsion Test ( "LSAT" ) , used in*law school 
admissions, and the (Graduate Record Examinations, used" ; 
by graduate schgols of various types/ Organizations of. ' . 
ed uca tor s and educational institution^, sponsor each-oil 

** * 

these tests. For example, the College 'Board, a nonprofit **** 
^nember sh ip <vgai)izat ion composed of collfeges, schools , 
school systems and education associations, .sponsors the <* 
SAT and the Law School Admission Council, a nonprofit 
membership organization 6t law school representatives, sponsors 
the LSAT, < ' ^ 



* 1979 Annual Report 22 {Educational Testing Service H 
1980 ), - 



' Coaching for Standardized Admission Te^ts 
Special preparation- for standardized admission tests, 
<£>ften referred to as "coaching," can take ittany forms,. Coaching 

'can be as simple *al& a few haurs' of . drill on test questions 

* \ ' - ' " ' 

patterned after > those expected to be t on the standard ized adm is- 

sion te*st. It can also be much more intensive 'and- complex 1 , 

including review "of tiik Substantive \conten^ jto J ^e tested by 

the exam, as well as instuction on 1 t£st-taking strategies and 

practipe on sample tests* Coaching programs are offered by .a 

variety* of organist ions , including commercial coaching 

schools] community groups, private tutors, and, in some cases, 

* *by secondary schools and colleges* " > 

■ TJhe Boston Regional Of f ice' s initial 4 investigation 

& into the coaching issue found that commercial coaching 

coyrses were being offered in every st^te and in vijrtuali-y 

* *■ _ 

every major *<:ity or college campus throughout the country, 
Tuitioo costs for such courses have ranged fron\ less, ^than 
$50 .to at least $300, Sdme of these .schools provide^no 
more than a weekend of instruction, while others provide 
'over forty hours of in-class time, as fwell as homework 
materials and practice tapes. K 

III. Importance of the Coaching Issue * * * , 

\ The questidn t of whether coaching for standardized admis- 

sion tests is effective, is importan^both to the students 

^^^^^ 

whpse abilities are being evaluated by the tests and to 
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the schools that ^se the tests as. -at least one basis by which 
to evaluate applicants fpr admission* Students concerned with 
maximizing* their chances for admission t<J the schools of their 
choige want tb know whether time and money would be ,well spent 
on spec ial preparation , Schools need to know as mjuth as possi- 
ble about the coaching issue in considering what weight they 

will give to standardized admission tests in making admissions 

'<■'-'• * * * ' * 

aftd placement decisions. As a practical matter, the more 

extensively schools rely on standardized test scores in making 

admissions decisions, the more important the questidn of 

coaching's effectiveness become?: ' .* ' 

1 Because the major focus of the investigation has been 

' 4 - / 

on coaching far the SAT, it is important t6 discuss more 

■ . • ■ - , / 

fully here the role that the .SftT plays in college admissions 



decisions. The SAT is taken by over a million students \ 
each year. It is part of the application process at hundreds 
of private and publiq colleges and universities throughout 
the country,^ The SAT consists of multiple-choice questions, 
with g^parate mathematical and verbal scores reported on ^ 
a scale 4 of 200 to frOO,^ For students in the 1978 graduating 1 



t 



Peterson's Annual Guide to 'Undergraduate Studyi 1980 
Ed it ion 37-59 (J, Hunter e&; 1979) thereinafter ^cited- 
as Peterson 1 s Guide ] , - 

i . * ■ * 

Educational Testing Service, ATP Guide for High 
Schools and Colleges 1979-81 ,, at 4, 8 (College Entrance 
Examination Board 1980), 



class, the fiftieth percentile raak was a verbal score 

* - $ > t-v 

of 430 and a mathematical score of* 470, * ^TS desfribjM* 

the SAT as measuring "developed vferbal and, ma^hfettat ical ' 

reasoning *abil^tie^j^iat ate # related to succ£ssful v performance 

io college, "5. i t $ purpose is said to be to, Serve as a 

predictor of academic performance in colldg^v£- i t 

The precise role which the SAT 'plays in^T^ecoLlege adftis^ 

sions process is currently the subject of ^ some controversy. 

Critics of %he> test are concerned that it plays too important*-^ 

* . ** * 

a role in those decisions, 7 In responding to such conceTps,*/^ 



Id, -at 15, , , , 

Id, . at 4, . 

Id, at 21, , The degtee t<y which , the SAT accurately * 
edicts college per f ormance has recently been the m 
subject of heated debate. Two recent publications 
Strenuously challenged the predictive validity of v 
the test, W, Slack fc.D, Porter, "The Scholastic . , 

Aptitude Test: A Critical Appraisal," 5*) Harv , Ed , * 
Rev , 154, at 164-69 <11980 ) ; A , Nair n i & .Associa tes , . 
The Reign of ETS: Tne_^Corp6f ajtiop^that . Hakes Up Hinds 
55-160 (1980 ) , Rex Jackson WT]$FS responded to 
the Slack and Porter article in^'The Scholastic Aptitude 
.Test: A Response to Slack and Porter's f Critical Appraisal 
50 Harv>, Ed. Rev, 382 (T980), to which, Slack and 
Por ter responded in "Training / Valid ity , and the Issue f 
of Aptitude: A Reply to Jackson," 50 Harv, Ed, Rfey; 
352 (1980 ), ETS respond ed «■ to Nairn ' s " criticism in 
two February 1980 pamphlets, Test Use and Validity 
and Test Scores- and Family Income , 

See f e ,g , , A. Nairn & Associates, The Reign of ETS:* 
fHe Corporation that Hakes Op Hindjs , 5-141 47-50 (1980 ) k 
Dean Vito Perrone of the Center for Teaching and Learning 
at the University of North Dakota ejc^pressed concern 
about undue reliance on the test because of the impact 

t . „. y, (Footnote Continued) 



ETS ftas*tw>fc£d that admissions officers have been advised by H 

' ETS and College Board ^that t^st* scores should not be the sole 
■d , * 

factpr used in making an admissions decision and that recent 
data indicates that colleges today are simply not very 
selective, 'that most/admit a large proportion of their 
applicants,** : * 

A 1999 survey conducted by the College Board and the , 

AineDicarv Association^ of Collegiate Registrars and Admissions 

* * 
Of fice-rs x sheds some light on the issue, , Of the colleges 

responding which ind ica£ecf\hat they did not have open- 

dfoor policies, only £,8% said that v admissions test scores 

were the single most important factor in making admissions 



(Footnote' Continued) 



on popr and minority students, , Truth in Testing Act 
of 1979; The Educational Testing Act of 1979: Hearings 
on HvR, 3564 and H, R, 4949 before the Suboomm, on 
Elementary, Secondary, and Vocational Education of 
the Hfrgse Committee on Education aryd Labor , 96th Cong , , 
1st Ses|, 455-5 9 (1979) (Appendix to the prepared statement 
of Vito^Perrone? Dean, Center fo'r Teaching and Learning, 
University K)f North Dakota), [Hereinafter references 
to the- record in the hearings are cited as dear ings 

,» on Testing ,] Paul Pottinger, Executive Diwctor of 
the National Center for the Study of Professions, 
expressed "concern ajDOut £el iance on; standard ized admi ssioji 
tests .an general: "Multiple-choice aptitude tests 
are unfsrir because thfcy fail to assess the more critical 
aspects rf of competence that .determine how effective 

\ students cbuld be as workers and productive citizens, n 
Hearings^on Testing , supra at 475, 

Educational Testing Service, Test Use and Validity 
11-12 (February 1980) , 



ttecisijOns, * However , 56 ,6% of those colleges cited admissions 

- test, scores as 3 ver^ important' factor^ and 33.1% cited 

them as^oae of several factors, 9 Perhaps more importantly, 

42% o£* the private fauP year colleges and 37,9% of the 

public four year colleges responding indicated that they 

r had minimum "SAT scores below which applicants generally 

are not considered eligible for admission, 10 

Other sources also indicate that a difference in a very 

* & 

small number of points on*tfie*SAT may, in some cases, be 
very important. For example) College Board's The Cpllege * 
"Handbook 1980-ffl noted the "Basis for- selection" a5 Flor id a 
State University was that in-state students mtist have a N 2*,0 ac 
ademic school grade average and a combined score of 800 on the 
SAT* for appl icants who had taken that standardized admission 
test, 11 Thk numbers for out-of-state applicants were an aver- t 
ag* of approximately 3,0 and a combined score of 1000.12 &t 
Arizona State University the "Basis top* selection" was said to 



College &oard< " November 1, 1979 Press Release, Table 
1 a. 



10 



4 



It 



"An Overview of s Findings from the College Board-AACRAO 
Survey of Undergraduate Admissions Policies, Practices, 
and Procedures" 12 (Prepared for the *1979 College* Board 
Annual' Meeting; New Orleans/ Louisiana* October 30, , 
1979), ■ ; 

The College Handbook 1980-81 , at 261-62 (M.Matheson 
ed . , College Entrance Examination Board 1980). 



12 id. 



be rank in top 50% of graduating clas^ or a- minimum SAT % 

combined score of 930 for in-*state applipants artd 1010 for \ < 

out^ofrstate applicants who take that*exam,13 ftn even more ^ 

complex use of minimum SAT scores is noted im th'e '1981-82 ' 

undergraduate application materials for yie University : - * 

of California, The materials include what is~ termed t;he T 

, * 

"University 6f California Freshman El ig ibil ity md'ex 11 for 

in- state appl iyn^rs , That index 1 is t$* the required gr;ade 

point average in certain specified course^ albng withythe 

required scores bn standardised admission te^sbs. For example, 

if the applicant has a 2, 79 grade point average, the required 

SAT combined score is 1580^ For a 3,01 gr*ade point average, 

the corresponding SAT score* is 1060, California residents 

with grade point averages of 3>3 or above are eligible „ 

for adm is s ion reg ardless of any test scores, ^ 

V 



13 'Id. at 34, : r 

14 University of California Undergraduate Application 
PacKet 1981^82 , at 13, 15, 4 Npn-residents mus^t have 
grade point averages of 3,4 or higher in the required 
secondary schotfl courses, > Id , at 16, ,The materials 
note* that the University recognizes that some students 
have t not' had. the &ame opportunities to prepare for v 
cdllege work and - thfe University offiets special admissions 
consideration for thosfe who*can offeer evidence of 

their ability,.to achifeye even though they "have not 
met traditional requirements, * The materials note 
that such appl icaticms 'are encouraged etfen though 
- i a limited member of persons may be admitted in that 
category. Id, IB, v 



10 



In assessing the role 6f SAT scores in coll.ege' admissions, 
it is al^o 4 m P ortant to consider tjie degree to which college 
bound students rely on their SAT scores in selec.tinci the 
schools to which they will apply. The answer to this question 
is not cleat!, 15 ^ 1978 survey of SAT takers notes that 
16% of the students responding indicated that their test . i 
scores had caused them to change their minds about the 
kinds of schools .they' were considering; in roost cases resulting 
in their lowering their* sights, 1*> Students clearly do 
have information at hand with' which to engage in self-selection 
on the basis of SAT scores. 

The score report sent to each SAT taker explains* his t 

f- 

or" her score and locates it relaf£ive*to -the scores, of other 
test tak*s,l ? In addition to whatever information on 
SAT scores the colleges provide in their catalogues, college 
reference guides include detailed information alsout the 
SAT scores of . enrolled *freshmen whenever* it is available, ^ 



15 r, Hartnett & ^ Feldmesser, ""College Admissions 

Testing and the -Myth o£ Selectivity: Unresolved Questions 
and Needed Research^," American Association of Hig_her 

« - Education Bulletin , March 1980, at 5-6* 

1$ Response Analysis Corporation, "SAT Monitor Program' 1 
- Al (July ,1978) . ' - 

17 . Educational Testing Service, ATP Guide for High'SghoOls 
and Colleges 1979-81 , at .13-17 (College Entrance 
Examination Boa^d 1980), % , / 



18 



Peterson 1 s Guide , supra n, 2; J t Cass* £ M.* Birnbaum, 

( Fodtnote Continued) 
* 

* / 
ii / 



.By* TeT review of such reference guides* prospective* applicants 
.can learn that at the California Institute of Technology 

i , t v 

one hundred percent of 1978-79 freshmen Mathematical scores 
o^the SAT were £00 or over,* 9 Over ninety-five percent of 
the 1978-79 freshmen SAT mathematical scores were 500 ,or 

over at the Rose-Hulman Institute of Technology, 20 Rice 

J 

University, , 21 Stanford University, 22 and the Colorado School 
of Mines, 23 At .Yale University, 24 Haverford College, 25 
Reed College 2 *> and Barnard College 27 over ninety percent 
of 1978-79 freshmen verbal sdores were 500 or above, - 
* The SAT is clearly viewed as an important part of 
the edir^tional system. Students do take their individual - 



18 ( Foptnote Continued) f 

\ v * 

. Comparative Guide to American Colleges (9th" 
* - e<J. 1979) . 

* 9 » Pete-rson's Gi*ide , supra n.2, at 40. 

20 ill at 53. ' 

21 Id. at 52. 
Id. at 55 . 

Id. at 42.. . * 

24 ili at 59. - ' 

25 ■ iii at 45 - 

26 



22 
23 



27 



Id. at 52. 
Id. at 39. 
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SAT scores seriously, 28 j n fact / .j^jjtcern has been raised 
that too many students may erroneously gauge their self- 
.worth or mental capac it ies /by -their SAT scores. 29 Secondary 
schools are very concerned aboftt the average SAT scores 
of ^ttje ir students , 30 Competitive colleges tend to use 
average SAT scores to evaluate the quality of their freshmen 
classes, 31 Communities view SAT scores as at least one 



28 



29 



30 



31 



See R. Moll, Playing the Private' College Admissions Game 
144-46 (1979) (Moll, who has served as admissions director 
at Bowdoin and Vassar colleges, recounts an instance where 
a high school counselor introduced a group of candidates 
with a list that contained the students 1 names 
and SAT scores, "No wonder candidates feel their 
scores ar£ tattooed on their foreheads. and it is their 
-fate to wander through life muttering at every turn/ f*» 
'I am a 510,' M )f J, Fallows, "The Tests and the 'Brightest 1 
The Atlantic , Feb, 1980, at 44, ("I have yet W meet , 
a hjgh school student who did not . take the tests as 
a measure of *hqw 'smart 1 he w$s,") An excerpt from 
the July 18, 1979 Chicago Sun-Times included in the 
record of the ^truth in testing hearings is also of 
note, "College entrance sxams have replaced the draft 
as perhaps the most important and mysterious rite 
of passage /for young people," Hearings op Testing , 
supra n,7,/at 666 , e. ■ 

W, Slack & D> PpiTter , ' "Tr air? ing , Val id ity,, and 
tire Issue of Aptitudet A Re-ply to J.acfcson," 50 Harv , " - 
Ed, Rev, 392, at 399 (1980), The authors are concerned 
by the use of the word "aptitude* in the name/of 
the SAT, "[Sltudents wha have interpreted low SAT 
scores as a reflection of their aptitude have been 
seriously wronged; their .capacity to learn has beer* 
disparaged ang the if sel f -esteem ^nd angered , " 

See S, Thomson & N, DeLeonibus, Guidel ines for 
Improving SAT Scores (National Association of Secondary 
School Principals 1978),. x 

R, Moll, Playing the Private College Admissions Game ,, 
146-47 (1979^, 
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indication .of the quality of the-ir school systems, 32 Nationally, 

as ap ETS researcher has noted, 11 the average test score 

Has become a kind of educational Dow- Jones index, "33 

I V , Federal Trade Commission Investigation of the Effect of 

Coaching on Standardized Admission Tests 

A, Focus' of the Investigation N 

The Boston Regional Office ("BRO 1 ') began its investiga- 

V 

tipn into the coaching is-sue when staff became aware of 

commercial coaching school advertising claims regarding 

the effectiveness of th^ir programs in preparing students £ 

to take a variety of standardized admission tests. These 

schools claimed to be able to improve students 1 scores 

i>y,as much as l00 points, JjtC ETS had. mainta ined that 

coaching would jjo^l ittle, to improve students 1 scores, 

-* 

BRO was concerned that students interested in improving 
their chances for admi ssion to the college or graduate 
school "of their choice were wasting considerable time and 
money in reliance on the coaching schobls 1 claims- of effec- 
tiveness. Pursuant to' Section 5 of the Federal Trade Comrois-> 
sion Act, 15 U,S,C, §"45, the Commission authorized the 

r - ■" 

+ - n 

L , " 1 : 



32 



S^e The Boston Globe , June 30, 1978, at 1, col, 2, 



33 " Donlon, "The SAT: Past and Present," The College 

Board Review ^ Fall 1979, at 29, 
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BRO to evaluate whether the commercial coaching schools 1 , 
claims were unfair or deceptive, 

r » 

BRO specifically examined twenty-two' commercial coaching* 
schools , focusing on , thoSe schools that offered preparation 
for the SAT and/or the LSAT, - St^ff evaluated voluminous docu* 
ment3 submitted by the schools to substantiate their claims of 
ef f ec tiVeneas i % In addition, in order to have an independent 
assessment of^the effectiveness of coaching, the BRQ undertook 
a statistical analysis o£ the effectiveness of commercial 
coaching fo/ b6th the SAT-'and the LSAT, ^ 

B, Findings of the FTC Regarding the Ifff ectivenessfof Coaching 
. The BRO analyzed an existing situation^ . compar irrg / 
the scores, of students who had voluntarily enrolled in 
commercial coaching coupses with a group of control students 

- i % " i * 

who had chosen not ■ to enroll in those dourses. Thus, the' ^ 
BRO study was not based on an* experimental design; student's, 
ye re not randomly. assigned to coached and control t groups v 
The coached sample, was developed from the enrollment lists * 
submitted by the commercial coaching schools. The control * 
sample was developed from ETS records of students who lived 
in 'the same geographical area and wjho had taken the same 
*administTation( s)* of the SAT or LSAT as the coached students, 
but who wer^ not identified as coaching school tenrollees. 

In September ofc 1?7B, the JSRO submitted, its memorahdum on 
the investigation bo " the Federal Trade Commission's Bureau of " " * 



Consumer Protection* (*BCP") in Washington, D,C, In its memo- 

* * ■ 

random/ BRO concluded that its statistical analysis showed 

■ ' « - i 

ihat coaching, was dramatically effective for the SAT* 34 

While the LSAtf was seen to be susceptible to coaching in a 'general 

sense, the results, showed areas where the effects were marginal. ^ 

The t BCP reviewed the BRO's memorandum and found that the 
statistical "methodology atilized did not sufficiently control 
for differences between the coached and control groups 
to permit the conclusions that had be4n reached, BCP undertook 
a refined* statistical analysis of the data that had been 
collected. The focus of t|ie reanalysis was to isolate 
the effect of coaQhing by controlling for a number of back- 
ground differences between the coached and control groups K , 
and by analysing the impact of self- selection, the fact 
that the coached ; students had voluntarily- decided to attend 
a cbmmprcial coaching course,. The BCP researchers concluded * 
that the Boston data contained enough background information^ 
ort the SAT test takers. to perform a meaningful; reanalys is , 
but that , suf f icent inf ormatibq -was not available for -,the LSAT> 

The BCP reanalysis of. the SAT data found .that/ coaching 
cbuld have a signif icant/positiye^ impact on scores * though 



3* ' Boston Regional .Of f ice , "Stajff Memorandum, on the Effects 
■ of Coaching on Standardized Admission Examinations 1 * 
157 (September 1978),* . 

.35 ia. " ( . • - . ■ . ' 

x 

, • 16 , ' 



the point value of the score improvement .found by the BCP 

was smaller than that found by the BRq, In the BCP report, 

the researchers concluded that coaching at one' of the two „ 

schools evaluated f was effective .in raising SAT scores by an \^ 

average of^appr.oA imately 25 p&ints .e^ch on the mathematical and- 

verbal sections of . the test, 36 ' f he/ students at the effective 

school tended to be underachievers fori stand-ard iied examinations, 

. ■ i * 

ancl thus the issu^ of whether other students might also 

benefit from coaching was not resolved in the report, 37 , 
The ver^ limited data sample that/ was available regarding 
students at the effective 4 school who were not und§jachi$vets 
indicated the possibility that they might also benefit 
from coaching ,38 



36 Bureau of Consumer Protection, "Revised Statistical 
Analyses of Data Gathered by Boston Reg ional , Of face 
, F qf the Federal Trade Commission" Executive Summary 
"(March 1979) [hereinafter cited as BCP Report ! ^ * 
Because of sample size limitations, the BCP analyzed 
* data from only two schools. 

Students who take the SAT for a second time/^e expected 
x to ex per ience score increases on average as^a- result 
of practice- and growth. When references are made 
in thiSr repor t to score increases &ttr ifautable to 
coaching, those increases are above and beyond what 
■would otherwise be expected from such practice and 
growth, * * 

Id „ As used in the BCP Repdrt, "ttnder.achiever" means 
a student who scores lower on the standardized .test 
than would have been predicted from socio-economic 
and- demographic background- data, 

* ■ 

38 .id. 



37 
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Th§ Commission recognized the importance of having 
the studies in. the public domain and- released both the 
BRO and BCP statistical analyses in Hay ol 1979* Underlying 
data^for the, studies; contained on computer tapes, were 
also raa^e available , through the National Archives,, ^ 

C, The Significance of the- FTC's Findings in Light of . ^ 
Prior Research J / ^ 

j 

* In ( order to ^ssess the significance of the conclusions 

' , v ■ , ■ * 

reached in the BCP report, BRO analyzed other research in 

1 * * * 

the area. While-each different study oiV the ' effectiveness of? 
coaching,, including the BCP study, must be evaluated in terms, 
of its own statistical design* and limitations, a number^of 
studies in addition to the feCP study do gpiht ->to the effecti^n 
of coaching for^t>e SAT, , 

Many of the earliest studies on coaching for theSAT were 
summarized by College Board *in a booklet entitled, Eff-ects of -, 
Coaching on Scholastic Aptitude Test Scores , copyrighted in • 
1965 and again in 1968, The booklet explained that the 
word "coaching" was used to refe£ "to a variety of methods 
used in-attempting to increase in a relatively short timeV 
students' mastery of the particular skills, concepts, and 
reasoning abilities tested by the SAT« W 3^ The booklet 



39 College Entrance Examination- Board , Effects of 

Coaching on Scholastic Aptitude Test Scores' 4 (College 
, Entrance Examination Board 1968) [hereinafter cited - 
K * as Effects of Coaching ] , ^ 

18 •. '* - 



included a lengthy statement *by ^!he College Board trustees 
issued as a result of , ttie studies testifying that "increases 
in scores on the SAT that may result from coaching are 
negligible. , 

.A recent article in the. Harvard Educational Review 
has questioned the. accuracy and completeness of that booklet, * 
The authors conclude that "there* is ample evidence that 
students can. successfully train for the SAT," and. that 
.such evidence was available at the time the College Board 
booklet was published in 1968. 42 



40 . .Id. at 8. 

41 T . ' W, Slack & D, Portal "the Scholastic Aptitude 
* ^ Test: A Critical Appraisal," 50 Harv, Ed? Rev, 154, 
at 15 5-*64 j( 1980 ) , / The authors assert that College 
^oard v failed to reference studies by Pallone (1961) 
and Marron (19^5), Id, at 158-5,9, The Harron studies 
i, involved hi^tt school graduates who spent seven months 
in private 5 preparatory schools. The studies did not 
include control gr6ups; Slack and Porter estimate 
the mean gains to have been 41,1 points" on verbal' 
Scores and 67,6 points on mathematical scpres over 
aijd above the gains, that would have been expected r 
frqm practice And growth, Id. at 159, The Pallone 
study involved one procjj;ajn of 20 students in a six- 
week; 90 minutes per day program of verbal instruction 
jand a 50 minutes per day verbal program for lb'O students 
from September-to March, Again/no control groups were 
^included. Slack and Rprter ^fetimate the jnean gains ' 
1 as 84,7 points op verbal for the gix-week- program 
and 79,1 points on verbal for the* longer course. 
Id, at*158-59. The six-Wfce(t Pallone study is discussed 
, Trv£&a at 23-24, ' 

42 * ia - at 164, As note<fc' supra at n,6, ETS has responded 
# ■ to the Slack and, Pprtfer article, R, <Jacks6n, "The 
Scholastic Aptitude Test; A Response to £>lack apd 
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Looking exclusively £t the studies s,ummarTzed by College 
Board, one finds that _for some groups of students and some 
forms of preparation score improvements of mor6 than 15 
points were reported. In one study, which involved tutoring 
in groups of two or three, coached students showed average 
gains on the mathematical section of the SAT of about 21 
points after approximately 18 hours of coaching and about 
26 points after 36 hours, 43 in 'another study! one group 
of students showed average gains of 18 points on the verbal 
section after preparation on special exercises developed 
by ETS to resemble items on the test, 44 Significant gains 
on the mathematical section were also reported in a third 



4 * (Footnote Continued) . 

" Porter's 'Critical Appraisal 1 ," 50 Harv, Ed, ftev , 
382 (1980), As part of that response, Jackson asserts 
that Slack and Porter failed "to draw clearly the 
critical distinction between programs of short-term 
drill and' practice designed to yield quick increases 
in scores, and longer-term educational programs designed 
, * to have lasting effects," Id, at 383, Slack- and Porter 
responded to the Jackson article in "Training, Validity, 
and the Issue of Aptitude: A Reply to Jackson," 50 
Harv, Ed, Rev, 392 (1980), 

43 ' Effects of Coaching , supra n,39>^at 16-21, The 18 

and 36 hours of coaching included instruction directed 
at both the mathematical* and verbal sections of the .test 

Id, at 14-17, . ^ 1 



r 



study foc^students not taking senior math, an average garin 
of abotjt 29 points over the control group,^ 

(Moreover, the studies summarized by College Board inay not 
^11* be relevant to an analysis of more sophisticated coaching 
courses off^^d to today^s'SAT applicants, A 1972 study 
sponsored by College Board noted two problems with earlier 
studies in discussing why further research on coaching 
for the mathematical section of the SAT was. needed. First, 
the instruction provided in the majority of the studies 
was, "where its nature can be ascertained, rather scanty. 
There* appeared to be little or no systematic attempt to ( 
identify the information and skills, needed to perform well ' t 
'on the test and tc^develop materials to meet these needs, 0 " 
Second, most of the prior research involved students at 
the extremes of the ability range rather than the more 
heterogeneous group o£ students who now ta^ the SATV^ 7 

The J972 study itself involved sevW'week, 21-ho*ir, 
coaching programs for high school j-uniors at twelve schools* ^ 

i 

Detailed lesson outlines, and other instructional material 



45 



Id, at' 12-14. 



L'. Pike & F. ' Evans, Ef f ects~~Pf""5ffe^'ial Instruction 

for Three .Kinds of Mathematics Aptitude Items 4 (College 

Entrance Examination Board 1972). 



47 
48 



Id.'.at Abstract. 
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developed by ET^ r were provided to the instructors Who 

were traced; iijf a twc^day workshop, 49 <phe authors concluded 

that students ^ould^be prepared with respect to each of 

'■ ■■ f. / ■ 

the threfe typ^s^Of math item formats evaluated in the study, 

+ ^ Ijtesyrlts of the statistical analysis showed that 
each of the three item formats was susceptible 
to the special instruction specifically directed 
toward it. The complex or nO%el item formats 

\ appeared to be more susceptible than the relatively 
straightforward item format. Female volunteers 
were found to "be slightly less able mathematically 
at the outset, and to benefit somewhat less from 
the instruction than male volunteers. Mean gains 

1 p£ h nearly a full standard deviation obtained 

by the groups instructed for the complex or novel 
formats were considered to be of practical consequence 

* and likely^to influence admissions decisions, 50 

Significantly, the results of the study were consistent for 
all twelve schools^ 51 One of the two authors, Lewis Pike^ 
later explai/ied ift,,a 1978 literature review of coaching 
studies that a judicious combination of instruction for 
the two -item formats used* at the time the 1972 study was 
conducted "would be expected to- yield an STI [short term 
ins trtfiction] effect of about 33 points, "52 1 



■ s 

49 -Id. ^at 13 and- 10. 



50 
51 
52 




* 1$.' at Abstract. 

. • .: ' ' • • • - 

U«»" Pike; Short-term Instruction, Testwiseness , and 
' the, Schplastic Aptitude Test: A Literature Review 
^ K With' Research Recommendations 16 (College Entrance 
/Examination Board Research and Development Reports', 
77-7,8, No, 2, January 1978) [hereinafter cited as 
Literature Review] , 
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In th4 1978 literature review, which was published as a 
College Board research and development report, Pike discussed 
and analyzed both negative and positive findings in prior 
studies' of short term instruction ("STI") for the mathe- 
matical ("S^T-M") and Verbal CSAT-V?) sections of the SAT. 
The abstract for the review no.ted thai: discrepancies "were 
generally resolved in favor of recognizing meaningful £>TI 
effects fior the SAT-M, but remain unresolved for the SAT-V, n 53 
Studies d.iscugsed showing positive results for preparation 
reievanJt to the verbal section did point to at least the 
possibility of meaningful verbal gains, in one study, conducted 
by Palltme, 20 students participated in a focused program 
of instruction on reading , vocabulary and logical reasoning 
abilitfes fbr 90 miriutes a day over six weeks, ^4 ^hose students 
experienced, an average gain of 98 points «on the verbal 
section of the SAT,^5 Because the study did not include ' 
a control group, Pike estimated the effect of coaching 
by subtracting the gain- one would have expected from growth 
^and practice. Pike estimated .that the instruction had 
produced approximately 60 point gfdns,^6 

: r ■ . - 

» 53 Id. at i. ? • ' 

54 , Id. at 19. . - » 

\ - ■ — / ■ ■ - 

.55 .id. . 

56 ' id. Pike used "the gains experienced by controls at 

(Fpotnote Continued) 

" 2i ^ 



In discussing the -negative and positive findings of the 
verbal ^studies reviewed,, Pike* noted that the tnos ( t marked dif- 
fererice ±jetween the, Pallone study and those with negative 
findings w^s the degree to which instruction in Pallone's , , 

study was subbantive and controlled , 57 pike also noted 
some -general considerations important to evaluating mixed 
research findings* Because of the importance of such considerations 
to an understanding of the research on 'coaching, they are 
^quoted here. 

Hote that, in principle, a single study showin^sub- 
stantial positive gains cannot be countered Or 
refuted by any* number of studies failing to get 
positive results. The only near -except ion would , 
occur in the ev^nt of a well-designed replication ' 
study that failed to show similarly positive results,* # 
In that case, there would be a discrepancy needing 
further study and resolution. Similarly, i"t would 
be fallacious to infer, from mixed results across 
studies on a topic such as STI effects, that across- 
study inconsistencies justify the conclusion that 
there are no meaningful effects. Mixed results 
0 'can flfean, as exempl i f ied _in Jacobs' (1966.) discussion 
of differences on ECT {English Composition Test] 
score changes from one experimental 'group to another,* 
that an effort should be made to find out- why instruction 
, was effective in some places but not in others, . 
This observation is particularly^True when making 
comparisons between studies in which little account 
was taken of either exam i nee *6r r instructional character is tics , \ 
A third observation is that there h&s been a' considerable 



56 (Footnote Continued) 
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the Bronx High School of Science as a rough (and probably 
conservatively high) estimation of control subject 
: gains Id, , t ■ f 

51 Id. at 60-61. ■ . 

. ' i ^ . '. 
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emphasis in most discussions of STI on the overall 
, magnitude of its- effects, with little consideration 
% given, especially when stating final' .concl usions , 

^ t6 differences among examinees, among STI curricula, 
o'r among item formats and other item char acter is tics . 5! 

- A recent paper, based on. research supported by the 

College Board , sheds further - light on thfe possibility of 

meaningful improvements from coaching for the verbal section' 

of the SAT, The researchers/ Alderman and Powers, evaluated 

special verbal preparation programs being offered in eight 

secondary schools , ^9 The instr uctional content of the 

various programs was not controlled by the researchers; 

the effect of existing school programs was measured, 60 

Most of the programs were extracurricular and only one 

involved a program of over twelve hours, 61 "Most schools 

* 

followed a commercial review book in their classes; one public 
i 

school { school A) distributed teacher -m^de mater ials intended 
for use in tutor ing individual students in a sel f- paced 
manner. "62 <phe estimated treatment effects on the verbal sec- 



58 Id, at 35-36, 

59 D, Alderman & D, Powers, The Effects of Special 
Preparation on SAT-Verbal "Scores , Abstract (College 
EntrSn_e t Examination Board Research and Development 
Reports/ 78-79, No, 4, February 1979)* 

60 " - id. 

61 Id* at Table 2, 
62 



Id, at 7 



25 



tion of the SAT ranged from +28.39 points to -2,75 points* 63 

* + Li 

The school at which the 28 point gain was- found was the 
schoof that used the teacher^made materials described above. 6 * 

Thus, the BCP findings regarding the effectiveness 
of coaching for both" the mathematical and verbal sections 
, of the SAT are not an isolated instance of positive find^ng^,/ 

Other research, particularly studies which focused on comprehensive 
, and better structured programs, also found positive effects 
x>i coaching for the SAT. 

D * SAT Descriptive Materials for 1979-80 
,In light pf prior research and the BCP's findings 
regarding the effectiveness of coaching for the SAT, the 
staff of the BftO met with ETS and College Board and Undertook 
a review ot SAT descriptive materials Regarding whether 

i 

the possible benefits of coaching had been properly disclosed 

4 * 

by ETS and" College, Board* The materials rev ieyed ^included 
pamphlrets for both students and educators * ^\ 

Perhaps the strongest criticism of coaching was an 
excerpt* from the College Board's Effects of Coaching on 
Scholastic Aptitude Test Scores booklet included in pamphlets 
for counselors and admissions officers in 1965 through 1968.^5 



63 Id. at Table~5, 

64 Id. ,at 7 and Table 5. 

65 College Board Score Report?: A Guide for Counselors 

■ (Footnote Continued ) \ 
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1 * * 

v* The excerpt was' the statement by the 1 trustees of'the College 
Board referred to 'at pages*18-19 supra * 

The trustees stated that "The evidence collected leads 
us to conclude that intensive drill for the SAT, either 
on its verbal or^ its mathemat ical^ part, is at best likely 
to yield insignificant increase^ in scores. The magnitudes 
of the increases which have, been found vary slightly from 
study to study, but they are^ always smJll and appeal to 
be independent of th? particular method of coaching used 
and of the level of ability of the students being coached. "66 
They expressed their belief that coaching would not'result 
rn important score increases even if conducted under different 
circumstances and with different students from those in 
earlier studies, 67 



6 5 '(Foojtnote Continued) 

and Admissions Officers 52-54 (College Entrance Examination 
Board 1965) [hereinafter cited as 1965 Guide ) ? College 
Board Score Reports: A Guide for Counselors and Admissions 
Officers 51-54 (College. Entrance Examination Board " 
1966 ) Thereinafter cited as 1966 Guide ) ; College 
Board Score Reports: A Guide for Counselors and Admissions 
Officers 56-58 (College Entrance Examination Board - 
196*7 ) [hereinafter cited as 1967 Guide ] ? ^College Board 
Score Reports: A Guide for Counselors and Admissions 
Officers 57-59 {College Entrance Examination Board - 
1968 ) Thereinafter cited as 1968 Guide], 



66 

* 7 , 1965 Guide at 53; 1966 Guide at 52-; 1967 Guide at 



1965 Guide at 52; 1966 Guide at 52; 19ft7 Guide at 
56; 196B Guide at 57, * 



57; 1968 Guide at 58, 
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Referertces- to coaching 'were harsh. The trustees noted 
that parents had "demanded tftat the schools divert teaching 
energy and time to a kind of drill that is obnoxious to ^ 
educators of every philosophy « "68 They closed their nine 
paragraph' statement by noting tttat they were most coacerrted 
because they saw "the educational 'process unwillingly corrupted 
in some schools to gain ends Vhich we believe to be not / 
only unwor thy but , iron ically , unattainable « " 69 

Though this harsh criticism of coaching was modified 
in iater years, even the L979-80 materials continued to 
indicate that special preparation was unlikely to be of 
sign if icant benefit « 

The student guide, Taking the SAT * simply explained 

that: v 

The verbal and mathematical abilities measured 
by the SAT develop over years of study and * practice « 
Drilling or last-minute cramming probably will 
not do much to prepare you for the test* However, 
if you are not taking a mathematics course, a 
review of mathematics concepts, such as those 
given in this bdoklet, will be useful,^ 



68 
69 



70 



1965 Guide at 53; 1966 Guide a.t SZ; 1967 Guide at 
56; 1968 Guide at 58, 

1965 Guide at 54? 1966 Guide at 53-54; 1967 Guide 
at 58; 1968 Guide at 59 (Footnote omitted in each 
reference « ) m The footnote explained that the statement 
had been prepared "for the information and adv ice 
of schools and col leges that had expressed concern 
about test coaching * n 

Educational Testing Service, Taking the SAT 3 (College 
Entr ance Examination Board 1978)* 
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Students were told that studying the sarflple questions, di- . 

rections and explanations provided in the guide, which 

***** " ■ " 

included a complete sample test, would help them prepare 

for the test* 71 * 

The Student Bulletin 1979-1980 addressed the coaching 

quest itfh in three Sentences , 

Before the test date, read the appropriate descriptive 
booklet, Taking the SAT or About the Achievement 
Tests , which will give you a better . understanding 
J of the test," Studies have shown that cramming 

does not raise scores, but knowing what to *expect 

can be helpful, A good night's sleep and a nourishing 

breakfast before you take a tes't are als<? helpful, 7 2 

The 1979-80 pamphLet for high school guidance counselors 

and college admissions officers, VTF Guide for High Schools 

and Colleges 1979-81 , d id ^include a lengthier distussion 

regard ing special preparation , The educator pamphlet stated 

that: 

Over the past 25 years, the College Board has ' 
conducted many studies, on the effect, of special 
preparation programs on SAT score results and * 
has supported the* independent investigation of 
the topic by others. These, studies consistently 
seem to demonstrate that "coaching," in the sense 
of intensive drill on sample test questions, 
does not lead to any "significant improvement 
in students 1 scores , Special preparation of 
other kinds 4 , however, may be helpful to .some r 
students* 

* / 

j 



71 — * " y\ 

72 -Educational Testing Service, Student Bulletin ■ L-979^1 9 8 
at 13 (College Entrance Examination Board 1979). 
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It is 'especially important for all students to - 
be familiar with the various types of questions 
in advance of the test. Students should know ! 
what the test is about and how it is structue^d, 
how to make the most ef f icient jase of time limits/ * 
how to "attack" the different kinds of questions, 
and when an "educated" guess using partial knowledge 
is sensible, Students w,ith such skills an^ knowledge 
about test taking are able to perform tQ the 
best of their ability* For this reason students 
should be . encouraged to study the mater ifil in 
Taking the SAT carefully and to complete Che 
sample test that is included , Schools/ may choose 
to assist students in the process through group - 
meetings &nd discussion sessions to ^inphasize 
the importance of this preparation, / 

Research continues to , show that a Review of mathematics 
concepts may be beneficial for students who are 
not enrolled in a mathematics coufrse at the time* 
the SAT is administered. Under /such circumstances , 
the review supplements previous/, classroom instruction 
in irvathemat ics , Some students/do lead lives 
without significant exercise in complex mathematical 
thought, and a review of mathematics' concepts 
may, therefore, serve to sharpen the mathematical 

■ t * reasoning ability of these/students , Verbal 

reason ing , on the other hand , is an inescapable 
" j , part of our daily lives; Mot surprisingly, verbal 

■ y - reasoning ability as me§£ured by the SAT has 

• * not been showrv to be as^responsive to similar 
instructional efforts ^ 

Given , the' BCP • s f indings £nd the pt ev ious stud ies 

which did find significant positive effects from coaching, 

/ 

the analyses of coaching contained in the 1979-80 SAT materials 

did not appear to fully apprise students or educators of 

bhe possible benefits of coaching. The Commission shared j 



73 Educational Testing Service, ATP Guide for High Schools 
and Colleges 1979-81 13 { College Entrance Examination 
Board 1979 ) (citations omitted) , 
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staf f ' s <^6ncerns about the apparent inconsistency .between ■ 
the SAT/m&ter ials and the research on coaching. Staff was 
authbr/zed *to bring these concerns directly to thfe attention 

ETS and the College Board and to offer thG organizations 




an opportunity to respond, t * • ■ 

Both organizations responded to the Commission' a. inquiry , 
submitted a copy of a i report on coaching now published 
in^a booklet entitled The Effectiveness of Coaehing for 
the SAT: Review and Reanalyis of Research from the Fifties 
to the FTC ("ETS Report").? 4 , College Board sqbmitted a 
memo it had sent to, the members of .the Cdllege Board,- as 
well as SAT descriptive materials for the 1980-81 testing year 
E, ETS Report 

' 4 

In addition to lengthy and detailed analyais of the 
BCP study, the ETS Report includes a review of studies regard- 
ing coaching for the SAT conducted before the BCP sfcudy, 75 



( 
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75 



Hessick, S, , The Effectiveness of Coaching/ fpr the 
SAT: Review and Heanalysis of Research frflm' the Fifties 
to the FTC (Educational Testing Service 1980S) [hereinafter 
cited as ETS Report ] , \ -/* 
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We will not here review the entire 135 page" report, but 

do wish to comment on a few rff the issues raised by ETS » 

regarding the BCP f s analysis* 

Many of ETS\s comments focused on the fact, that the , 

BCP study was not based on an experimental design* 7 ** 

As noted supra , the BCP did not, identify students and thin 

randomly assign them to a group that would be coached and 

a group that would, serye as a control* Rather, the BCP 

evaluated students who had voluntarily sought out commercial 

coaching and compared these with a control group* While 

ETS acknowledges that* the BCP's analytical design did control 

for a number of background fafctors that could be associated 

with self-selection, EtS emphasizes, that it is impossible 

to know and control for all background differences and 

all possible impacts of self-selection* 77 
■ 

ETS argues that the impact of self-selection is most 
likely to mean that the BCP f s results overstate the ef- 
fect of coaqhing, that whatever f actorj^otivated students, 
to take a coaching course malce it likely that such students 
would have dope better on the SAT even in the absence ^^Q.f^^ 
coachihg* 7 ^ The Report does refer fco the possibility^ that the 



76 S ee , e *g. ETS Report , supra n*74, at 3 and .33 * v 
1\ ETS Report *, supifa n*74, at 32-33** 
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Id* at 61* 



impact of self-selection causes not controlled for by either' 
the 3CP or BTS analyst could be to underestimate the actual 
effect.^. ^ # ? 

In considering the self-selection issue', it is^importaht 
!to ( remember , that, in addition to ha^iog controlled for £ 
variety of background factor's such/as grades, and^eported 
family income, the BCP examined the data for/ the possiblil^r * 
of one ^likely reason that students would $eek ^commercial 
coaching - whether the coached students had performed^ follow 
their expectations on pr.ior testing, 80 ^ e concern was 
that "the "lower than expected .prior test scores might be 
a statistical accident affid^that such students would thus^ 
perform better orT^s^^quent Resting regardless of coaching. 
The BCP's analysis did indicate that the coached students 
at the effective " school wef e under achievefr s on standardized 
tests, but also found that t;hose students would probably „ 
have continued to underachieve in the absence of coaching , 
fhus r the BCP concluded that its estimates of coaching 
eff ectfe , approximately 25 ( points on e&ch of the two sections 
of the exam, did 4pt have to be adjusted, 81 ^ * 



79 -V id, at 59, * . 

' 8a BCP Report , supra n.36, at 2'4-34 
81 BCP Repo r t , supra n.36, at 34. 
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ftfter .the general commentary on the BCP study, the 

ETS Report presents "the result^ of two marjor statistical 

reanalyses undertaken by ETS, One, conducted by T^W.F, 

Stroud* used a statistical model similar to the BCP model 

but wit'h certain additions and modifications, and also included 
- . ; # V 

data from' a third coaching '^school which the BCP had not 



»* evaluated* because of its small sample sizS^T*^ Stroud also 
investigated the interaction between the size of the coaching 

* m s- 

effects at the two largest coaching schools and 'certain 
- background data,^jCh as race and income, ^ 

**The Stroud analysis Confirms the BCP's estimate of ' 
effects, Stroud found what he termed "combined coaching/self- 
selection" effects of about 20-33 points for the mathematical 
section and 27-34 points for -the verbal section for students 
at the effective school. 84 These results closely parallel 
the BCP's findings .of an average coaching effect at that^ 
school of approximately 25 points on each of the two sections 
of the test, Stroud found riegligible and inconsistent * 
-effects at the other two schools, 85 . At the larger of those 



\ 



83 



, ETS Report / supra' n.74, at 3, . 

Id . at 3*. The Strou<3 analysis is discussed 

at pages 43-S1- of the ETS Report and set forth in 

Appendix 2-, ■ ' 



84 ' Id. at 46. 
85 id. at .46. 
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two schools, Stroud did find interactions between both 

race and reported parental income and the "cbmbined coach ifcg/sel f~ 

se^tfction" effect on the verbal section of the exam. 86 

* The second Preanalysis, conducted by Dbnald Rock, applied 

* 

a statistical model whiph is designed to take into account - j 

any differential Kates of growth in SAT scores for the 

coached and control groups. 8 ^ He identified groups of 

coached ( at the^ef f ective school ) and control stud^JJts 

for whom scores' 1 were available oti the Preliminary Scholastic 

Aptitude Test ("PSAT") and two administrations of. the 



SAl\ 88 The growth rate^ beween the PSATand the first 

f 

SAT were detrerm ined *f or both groups.' Rock found that the 
coacWa and control groups were growing at virtually, the 
same rate' on the mathematical section of the tests, but 
that the coached group was growing faster than 1 the control 
group regarding the verbal Section. 8! * Rock assumed that these 
growth rates, absent coaching, would continue unchanged be* 
tween the first and second 1 administration of the SAT* 90 The 

86 Id. at 47-51. \ r + 

87 id. at 3-4. The Rock analysts is discussed , . ' * 
at pages 53-59 of the ETS Report $nd set forth in 

Appendix 3. - . • 

88 • id. at 54. ' .' * 

89 id. at 5.5-57. - ' 

90 I 3 .- at 57. ** • . \ 
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.growth tft$del was then used to estimate the coaching effect 
while ^controlling fox the difference, in growth rates, 
Rook found thjat if one assumed " that the coaching took place^ r 
on average, midway between the first and second administrations 

V 7 - , 

of' the SAT, the effects were approximately 30 points on 
/ thl mathematical scores, and* 11 points on the verbal scores. yi 
. However / if one assumed that the* 'coaching took place immediately 

af&er the first administration of the SAT, the effect was 

about 31 points on mathematical scores' and , 17* points on 
verbal^52 

Rock^noted that it was more, likely for students to 
* r * 1 , i ■ 

obtain coaching as close to the second administration of the 

SAT^Ts passible, and 'thus ooncluded that the 11 point figure 

V * ■ - 

t was jy'otSably more accurate* than the 17 point, figure in 
assessing the^erbal effect, 9? In discussing the Rock 
Analysis in the Overview of the ETS Report and in a paper 
entitled "The Ef f #o*<veness of Coaching for the, SAT: Review 

and Reanalysif of Reseach from the Fifties to the FtC \ 

, j * > , 

Summary of Issues and Results 11 d^^egiinated at the end 

'/ - - 

of May, ' ETS disctfssed only Rock's 11 point finding regarding 



t - 



91 



■. Id:'«a£ 57-58 . • . (_ ( j 

* 2 * Id. <it ; 57-58. 

^ 3 S Id. a^ 58, .132. - ' • 
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. the verbal section of the te^t , 94 The assumption about 
the tim ing of the coach ing/'seem's reasonable since students 
are likely to believe thj&t the closer the preparation is 

^ to the test *in questioj^,- the more likely brte is able to 

« best' util ize the training. But, the assumption makes it 
difficult to understand why the effect was found to be 
smaller if the coaching took place closer in time to the 

v te&t in question. 

Substantial efforts were made fry FTC staff and consultants 
to replicate the computations that led to Rock's 11 point 
finding. Because these efforts were unsuccessful, the 
BRO requested an explanation of and support for the 11 
point finding, t Shortly after the BRO inquiry, ETS published 
an Errata Sheet concerning the Rock analysis. In the Errata, 
ETS abandoned its reliance on the 11 point finding and 
adopted instead the higher 17 point finding* for the verbal 
section of the test. Because the FTC's efforts to replicate 
the 11 point - finding were unsuccessful and publication 
of the Errata came shortly after the BRO inquiry, Commission 

t staff concludes that the 11 point finding was due to^a con- 
ceptual ojr computational error . 



94 Id, at 4; S, Hessick, Whe Effectiveness of Coaching 
for the SAT: Review and Reanalysis of Research from 
the Fifties to £he FTC - Summary of Issues and Results" 
5-6 (Undated) , 
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In, considering even the 17 point finding, at least 

ft 

one observation* should be kept in mind. The Rock analysis 

is based orr the assumption that the coached group, who 

did start out with higher scores than the control group,* 

would have continued to grow at the same faster pace absent 

coaching. While the assumption of linearity is not unreasonable, 

it is impossible to verify its accuracy, ^5 n- £ s at least 

possible that in the absence of coaching, the coached groups 1 

1 

learning would have peaked or at least tapered off between 
the first and second administrations of the SAT, in which 
case Rock 1 s 17 po int f in<3 ing under est imates the coaching 
effefct,. Finally, regardless of whether the verbal effect 
was the 17 points found by Rock, the 27-34 points found 
by Stroud o^" the approximately 25 points found by the BCP , 
the effects estimated through all the analyses are significant* 
F, SAT Descriptive Materials for 1980-81 

As noted earlier", the College Board provided the Commission 
with copies of the new descriptive materials on the SAT 
for 1980-81* Of most significance is a new statement on 
coach^f^, distr ibuted to secondary schools this f all , entitled 
"Six points about Special Preparation for the SAT* A Message 



^5 other researchers have commented on possible problems 
prSsented^by the linear growth, model « * A* Bryk, J, 
S treni</"& H* Weisberg, "A Method for Estimating Treatment 
Effects When' Individuals *are Growing* 1 ,* 5 J« of Educational 
Statistics 5, at 25-28 (1980), 1 



38 



<to Students** (**Six ^Points Message**),. This message is a 
much more comprehensive Statement on coaching than has 
-been provided ta students in the pafet. It makes clear 
^that there are different types of special preparation offered 

^ indifferent settings and acknowledges that different results 

/ 

may occur. The message includes the following discussion 

of studies on the effects of coaching* , 

S tti'd ies of special preparation programs car r ied on 
, in many high schools show various results averaging 
about 10 points for the verbal section and 15 points 
for the mathematical: over and above the average increases 
that would otherwise be expected f ronTintellectual 
growth and practice. In other programs- results have 
ranged from virtually no improvement in spores to 
average gains as high as 55-30 points for particular 
s groups of students or ^ar^ticular programs. Recent 

studies of commer cial coaching have ^hown a sim ilarr / 
range off results , 

Thus, while the message still presents the average of effects 

found in studies of some number of high school programs, 

it does go,, on ^to note that studies of other programs have 

shown greater increases* Perhaps most important is the 

fact that the message .acknowledges that improvements^ as 

high as 25-30 points on a section of the SAT are possible* 

While the statement may not reflect the very upper limits 

s ■ » 

, found in studies such as Pallone, it is far superior in 
its breadth and detail than earlier statements made in 
'the SAT descriptive materials,, 

r 

Unfor tunately , statements regard ing coach ing in the ' 
1980-81 versions of neither the bulletin nor guide for students 
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nor the guide for counselors and admissions officers parallel 

the Six Points Message* These materials ^ppear to lump 

t * 

previous studies together to come up with a 10 point coaching 
effect for verbal and 15 points for math,^ Even if these 
numbers werfe "produced by a fairly computed averaging of 
all prior studies, the number tells students an^ educators 
little* Students who wish to properly prepare themselves 
for college admissions are not interested in such statistical 
averages* While they have a need to know that some formfe 
of special preparation have been shown to be of little 
effect, they have a right to know that other studies have 
shown significant positive improvements. It is essential 
that students be provided with enough information to make 



96 



Educational Jesting .Service, Student Bulletin 1980-1981 / 
at 12 (College Entrance Examination Board 1980); " 
Educational Testing Service, Taking the SAT 3 (College 
Entrance Examination Board 1979 ) ; Educationa-1 Test ing * 
Service, ATP Guide for High Schoots^and Colleges , • 
1979-81 , at 13 (College Entrance Examination Bo£rd 
1980)* The 10 point verbal effect and 15 point, math 
effect may be based on only certain coaching" studies* 
For in summarizing prior research, the ETS Report 
notes that "the average coaching effect acro^£ studies 
having some type of control group was less than 10 
points for SAT-Verbal and le^ss than 15 points for 
SAT-Math** ETS Report , supra n*74, at 29, These 
estimates do not include the BCP findings or the finding 
of studies which did not have control groups* with 
respect to the latter, the ETS Report states: "The 
provisional estimate" of average program effects for 
these noncontrolled studies was 38 points for Verbal 
and 54 points for Math** 1 ETS> Report , supra n*74, 
at 29* * 
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an intelligent .decision about whether ^o^seek special preparation. 
Similarly, but perhaps even more importantly, it is essential 
that educators who rely on the test to advise and to make 
admissions decisions be 'fully informed about this important* 
topic. 

By letter to staff of the Commission, College Board 
has indicated that it intends to revise the bulletin for 
students and other appropriate SAT materials for the 1981- « 
82 testing year to include the Six Points Message or its 

equivalent. It is hoped «that in making future revisions 

v 

the College Board continues on the course toward full information 
begun by the Six Points Message, 

V, Implicat-ions of the Federal Trade Commission Finding^ 
on Coaching 

In the course of the investigation* especially in light 
of the results of the BCP study, it became clear that many 
educational policy questions are raised by a finding that 
coaching for a standardized admission test such as the SAT 
can be effective. Perhaps 'the most fundamental question is 
simply "Why?** The research conducted to date does not provide 
an answer to the que st ion , The focus of coaching studies 
has 1 been on whether coach ing is effective rather than" why « 

In its recent report, ETS recognizes that a finding 
that coaching could subst£nt ially iirrprove scores would 
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have important educational implications,^ e*es also recognizes 

the question of "why" coaching gii9ht* work &s an important 

one , ' * ' 

One 4cey issue is the extent to which increased test 
Scores attributable to coaching may represent stable 
long-ternr improvements in the verbal and mathematical 
reasoning skills measured by the SAT pr instead, reflect 
the overcoming' of inadvertent sources of test 'difficulty 
unrelated to these reasoning skills, such as difficulty 
associated with test anxiety, and unfamiliarity with 
different item formats and test-taking strategies,^ 



As ETS sees the issue, the question is whether the score im- 
provements reUate^to the substance of the test or the test's 
construction « u 

Lewis Pike, in the 1 97 B 'literature review discussed supra 
at pages 22-2 5 r notes in somewhat more detail the possible 
ways that coaching could affect scores « ^ Through instruction 
regarding the basic skills that the SAT is designed to * , 
measure, the student could be better prepared to demonstrate 
the under 1 ying knowledge and reasoning abilities; he/she 
has already acquired during years of formal schooling, 
Though students may have a wide base of knowledge of those 
skills being tested, coaching might help to improve scores 



9' ET6 Report , supr a n.74, at 4, 61-67. 
98 

99 



98 Id. at 4. 



Literature Review , supra n.52, at, 71. 
100 Id . 
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by filling in any gaps between that underlying basje and 
the actual test .contented Instruction in testwiseness 
skills could help the students achieve a better score by 
familiarizing them with'the Ttem formats, and by explaining 
test str ateg ies , such as when to guess on the basis of ■ * 
limited information* 102 Coaching, could help by reducing students' 
anxiety level or by helping them to cope efficiently with 
a timed test format * 10 -3 

At least one other explanation for possible scor^ 
improvements is that coaching could teach students how ^* 
to answer questions $n. the test without any understanding 
of the skills which are .being tested* For example, a test 
might contain ex tranequs clues that the stadent could use 
to select the correct answer* However, as ETS notes in 
its report, in a well designed test such .extraneous clues ^ 
should be rare* 104 

The answer to why coaching 'may be effective is important 

in determining the implications of coaching for test validity. 

First, what are the implications for validity if coaching is 

* 

improving the skills the SAT is designed to measure? in ETS 1 s 



101 Id. 

102 id. 

1° 4 ETS Report , supra n.74, at 65. 
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view, if any improvement in scores is related to long-term 
improvements in the underlying sk illy being measured , then the 
possibility of such improvements does not invalidate the 
SAT, 105 But, such improvements nfay say something about the 
skills being measured by the test. The authors of the 1972 
coaching study discussed supra at pages 21 afrtd 22 addressed 
this issue, "(Hf specials instruction were found to influence 
SAT scores substantially, the validity of the test would be op$n 
to question, since it/ is intended to be a measure of relatively 
stable at tr ibutes developed over a long per iod of time . "106 

Skill related coaching whi^h affects something other than 

/ 

long-term improvements should also be considered, if coaching 

/ 

is serving 6 review function, then a well reviewed student's 
score is/likely to be a better indicator of the student's ac- 
tual potential ,10 ? On the other hand, if the student is learn- 
ing the skills being tested, but retains them only long enough 
to perform well on the test, the test's ability to predict fu- 
ture academic performance may be reduced. 

Also to be considered are impl ications for test V 
validity if coaching can help students to improve -scores 



, 105 Id, at 4, 62-63, 

10 *> /L. Pike & F. Evans, Effects of Special Instruction 

for Three Kinds of Mathematics Aptitude Items 4 (College 
Entrance Examination Board 1972) (citations omitted). 

Literature Review , supra n,52, at 71, ^ 
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without a concomitant improvement in underlying skills. 
Alderman and Powers noted £n their 1979 review of verbal 



coaching programs two possible implications, 

: To the extent that special preparation foj: an aptitudW 
test leads to scorj$;/£b&nges unrelated to the criterion 
( e ,g-, , college graS^ypoint average or^ofe performance), 
that preparation actually impairs thr£test!s usefulness 
as a -predictor of later success, v On the other hand, 
when a test contains unusual or unfamiliar- item types, 
special preparation may help students in attaining 
scores more indicative of their ability, 108 

The question the^i is whether instruction in testtwiseness 

is permitting the students to somehow obtain inflated scores, 

or simply to better demonstrate their underlying abilities. 

In the first case, validity is questioned A® 9 Iri th4 second 

case, at least'for those students' * scores, it is not, HO 

In assessing the question of test val id i ty , however , 

it is important to consider not only the validity of the 



108 D,iAlderm?n, & D, Powers, The Effects, of Special * 
Preparation on SAT-Verbal Scores 1-2 (College Entrance , ^ 
Examination Board Research and Development Reports,, 
78-79, No, 4, February 1979), 

109 The ETS Report notes that "if improved test wiseness 
leads to test scores that are inaccurately high, the, 
interpretive validity of the test would be diluted 
and its'predictive val id ity jeopard ized , " ETS Report , 

- supra n,74, at 63, 

110 *phe ETS Report not^s that: "I^f improved test wifeeness 4 
increases test |*tfres that were inaccurately low because 
of construct* ivfe levant difficulty, a more accurate 
assessment of ability level would result and the predictive 
value of the test should be enhanced*" ETS Report ; 

supra n , 74 , at 63 , 
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cbached' student' s score, but. also the validity of comparing 

the scores of different students* Is it appropriate to 

compare the score of a coached student with the score of 

an uncoached student? The question is particularly 

important because the materals reviewed during the FTC's 

investigation indicate that all students do not have equal 

access to -xxtaching * 

As noted supra , tuition costs at commercial coaching 

schools can be high* with costs of up to $300 for an SAT 

course, many students are clearly priced out of that market* 

The BCP's analysis of the coached and uncoached students 

~~ tfn Lts study indicates that it is the more affluent 

"studentythat is likely to bV coached at commercial schools* 

The BCP found a .statistically* si^nif leant difference in 

the parental ^incdmes of the coached and uncoached student 

groups it evaluated . ^\ Over forty percent of the coached 
* 4 * 

students reported parerffcal incomes of $30,000 or more, 

while the comparable figure for the uncoached students 

was only slightly over seventeen percent«H2 Further evidence 

of the lively spcio^ecortomic split between coached and 

uncoached SAT test taker^ is found in the 1 BCP's analysis 

of the type of high' schools the students attended. Approximately 



HI . BCP Report j supra n.36, at 8. 
112 Id . 
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S6venty-f'ive percent o£ the uncoached students attended , 
r - \ ^ . 4 

public schools, -(/hale only about fifty-five percent 'of > . 
the coached group were public school , students ,H3 . „ , 

We also suspect that secondary schopls with a sm^.1 

* . * v , 

percentage pi college bound .students are lestf likely to # 

have the resources to offer test preparation courses to' 
- their -students, 4 Healthier high schools may l^ve the * * 

* qapacity to offer precisely the type of intensive review 1 
that h§s been shown to be effective in raising scores, while 
, financially pressed schools may not, 

^ The' question of Access takes on added significance if- 
whatever benefits, coaching confers are dependent, on the charac- 
teristics of the student^ themselves,. For it is at least pos- 
sible that th4 Students molt' in need of ^raini-ng may haVe the 
least access to it, Fpr ' example ,\if instruction in 4 
testwi^bnesS skdlls is effective, the students most in n^ed'df 
such instruction ^ay be those who have, had the least' exposure 
ta multiple-choice format examinations prior to taking the V 
SAT, * Students at schools in wealthy neighborhoods are more, 
likely t^an the^ inner city scRool counterparts to have been 
expo^d'ij^a jiuffifew of multiple-choice standardized tests ^ 
before taking the SAT,U 4 * # 



; H3 .id, at 9, ' • r 

114 Hearings on Testing , supra n.,7, at 687 (Testimony 
P ■ * " " ^ - * {Footnote Continued) 



The Sttoud analysis findings of an ( interaction between 
^ reported parental income and 1 the "combined coach ing/jelf' 
selection" effect on the verbal section of the SAT also 
deserve furthet inquiry. The ETS Report suggests two possible v 
J explanations for why students* reporting lower parental 
income experienced greater* score ir^creases^ after attending 
a coaching program ^than did those reporting higher parental 
1 income/ One of the explanations suggests that economically 
disadvantaged ^students h^ve the most to gain from such 

* coaching , * * 

, i 

\ Since sjtudents with low 'family incomes may have Had 

* less ^ccess to learning resources at either home* or 
school, this interacticm might have arisen because 
some of them, chose to attend coaching school tfs a 

• limited-cost effort to gain short-term intensive access 

to such resources, which, they then used to good compensatory 
effect, 115 / . 

1 The rival hypothesfs referred to the self- select ion 

* 'issues? £haT motivation was responsible for the score in- 
creases rather than the cqaching program* itself, * 

[S]ince both motivation and financial means are probably 
instrumental in deci'dihg whether or not to attend 
> a commercial coaching school^ it is not * unreasonable 

that coaching school enrbllees having 'lower than average . 



114 (Footnote Continued)* 

of Althea Simmons / Director , Washington'Bureau of 

the Nafeotial Association' for the Advancement* of ' Colored 

people) , + \ ' • 

* * 

■115 ETS % Reports supr a n/74, at 48, 



financial means ^might have especially strong motivation 
to perform jtelilH* 

Stroud's finding of an interaction between race and 
the "combined coaching/self-selectlon" effect on verbal , 
scores, where the "coaching/self -selection" effect was* 
larger ^£or blacks than for nonblacks, also raises interesting 
questions about differential coaching impacts ^ Whil*e 
the sample size- evaluated was very small, the report notes 

h that becaiise of the si* of the effect and its statistical 
significance, further research is clearly necessary 

In a cecent presentation at an American Psychological 

* Association symposium, Lewis Pike presented his analysis 
of the equity of access issue, He noted that when coaching 
works,. and there is unequal access, a profound unfairness 
is done to the students without access. He also noted 

*T*that for students who were ^coming from already disadvantaged 
educational backgrounds, any lack of prepajration-in how 

- to cope with a test constituted a -double jeopardy. 



117 Id, at 3, 48-51, 
Id, at 51, 

119 paper presented on September 1, 1980, in Montreal, 
Canada, at the American Psychological Association 
Symposium -on Truth in Testing Legislation, Mr, Pi-ke, 
who had served as*an ETS researcher for' a number of 
years, is now with the National Institute of Education 
of the Department of Education, * 
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ETS has also commented on the equity of access issue, 
making some distinction on the basis of^what the coaching 
program is accomplishing. If a coaching program could improve 
underlying skills, thenjthe issue of equity of access 

■■ *' / 

"is tihus j similar to the issue of equity qf access to < 

r * * 

effective school programs or effective life ^cperiences. 
Thus, coaching or special preparation programs producing 
increased t£st scores by improving the abilities measured 
, , would have important . implicatidns for educational practice 

and social policy, "120 jf ^ coaling program could be^ effective 
because of" "the testwiseness ttijairuing given, then the issues 

* . - /* - • 

of equity of access are also important, but the implications 

revolve aroqnd Resting practice, ETS */puld then recommend 

• * 

that any difficulty related to the* format of the test be 1 

reduced and that /substantial efforts be macte to increase \ . 

test familiarization and testwiseness, 121 ^ 

x The final issue then, wjaich ETS 1 s comments have raised \ 

is what steps* should be taken if . coaching is effective ; 4 

but access is fcot open 4 tQ ^11 students, Before pursuing 

the options that ETS has sugge^ed; >or others, it i£ first - 
, • \ « . . . ■ 

^important to fully understand the.preciise skills and knowledge 

that the SAT is in fact testing^ to; evaluate 1 * the' value* * 



120 ' ETS Repo-r t , supra n.74 / at -64-651 

121 "id." 65-66. 1 
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of developing such skills and knowledge, and to evaluate 
the value of training students to demonstrate those skills 
on any particular examination ,1 22 Fur ther research and 
analysis are clearly necessary before answers can be found 
to the many im|>ortaht educational questions raised by a 
finding that^coaching can be ,ef fective, 
VI , Concl usion 

The purpose of the FTC's investigation has been to 
assure that the various organ izations „ involved in the standardized 
testing industry accurately explain the effectiveness of 
cdaching 1 Progress has been iqade and staff is continuing 
to monitor the market to see that full and fair disclosure 
becomes the norm , 

Ir\ the course of its investigation, the FTC staff 
ftas produced an- important study regarding the effectiveness 
of commercial coaching for the SAT. However f the investigation 
could not and has nbt attempted to resolve the many educational 



1^2 These questions impl ic i tly include questions regarding 
the predictive validity of the test noted supra at 
*n . 6 and the possibil i ty of rac ial,, cultural o**^ economic 
bias. See , e , Hearings on Testing , supra n.7, 
at 545-55 (Statements of James Loewen, Director, 
Centfer for National Policy Review, School of Law, 
Catholic University of After ic a) , Further , the value 
of muTtiple- choice aptitude artcT^^hievement tests 
has been cr it ic ized ^because such- tests *3o no*t measure 
motivation; or creativity, and concern, raised about 
Whether- they properly recogni ze v the talents of the 
deep and subtle thinker*, B, 8pffman,. The Tyranny^ 
■ + o£ Toting 91-101 (1962 ), ■ 



policy questions raised by a finding that coaching for a 
standardized admission test such as the SAT can be* effective, 
Researchers have only begun to address these questions * 
We have discussed some of them here to encourage experts 
involved in the field to invest Jtheir expertise and resources 
to f ind appropriate answers, v 



r * 



r' 



52 p 



ERIC 



